anyway you can limit this to english only? I tried this branch and it filled up my disk.

anyway you can limit this to english only? I tried this branch and it filled up my disk.

For english only just limit the language list here to contain only "en": https://github.com/lifeiteng/vall-e/pull/111/files#diff-9c086567a8bee92cd4ae661ae5d75be66ae5340f8982cb287a09a78aee2041bdR22
You might want to comment out this lines also: https://github.com/lifeiteng/vall-e/pull/111/files#diff-9c086567a8bee92cd4ae661ae5d75be66ae5340f8982cb287a09a78aee2041bdR116-R125
And remove the _subset from the test / dev datasets, too, so you have a bigger test and validation dataset: https://github.com/lifeiteng/vall-e/pull/111/files#diff-9c086567a8bee92cd4ae661ae5d75be66ae5340f8982cb287a09a78aee2041bdR134-R135

Thanks! I will give it a try

Inorder to get reasonable result, we need design the multi-language Symbol set, work with Language ID.

@RuntimeRacer please update prompts.

Can you share the results here?

@RuntimeRacer please update prompts.

Can you share the results here?

@lifeiteng I just started training the NAR model today; I will share results in a bit once first few epochs have been completed.

Will also update comments then.

@RuntimeRacer - any updates on the training performance?

@pawel-polyai It's currently training NAR Epoch 4 on 6x RTX 3090, after training 10 Epochs for AR; Intermediate Results are mediocre so far; it is able to Synthesize Speech (Tested only English and German), however it is still not able to fully maintain Speaker Identity nor accent.

For Example, after NAR Epoch 1 it spoke with seemingly slavic accent; After 2 and 3 it Changed to French for some reason; so not sure how precise it can get yet, nor if the Accent it is speaking with is coming from an attempted transfer of Speaker Identity, randomly based on last trained Training data, or Dataset Bias.

However The Loss is still decreasing for NAR and I'll keep you updated. Sharing Traning graphs here as well:

Also sharing my (very not-in-depth) Examples; Only tested with one speaker which I found TTS Models to have a hard time replicating in the past; and also the intermediate models NAR Epoch 1-3: https://drive.google.com/drive/folders/1-bCwvXdXd4O2NOBigoXVdArAnZoigvWc?usp=sharing

If you want to play around with it yourself, you can perform inference with these commands:
python bin/infer.py --output-dir ./exp/valle/results --model-name valle --norm-first true --add-prenet false --decoder-dim 1024 --nhead 16 --num-decoder-layers 12 --text-prompts "You're in the presence of Suzette!" --audio-prompts ./prompts/Suzette_crop1.wav --text "I am the magnificent dark princess of the netherworld." --checkpoint exp/valle/epoch-3.pt

python bin/infer.py --output-dir ./exp/valle/results --model-name valle --norm-first true --add-prenet false --decoder-dim 1024 --nhead 16 --num-decoder-layers 12 --text-prompts "You're in the presence of Suzette!" --audio-prompts ./prompts/Suzette_crop1.wav --text "Ich bin die großartige Prinzessin der Ätherwelt." --checkpoint exp/valle/epoch-3.pt

Thank you for your detailed and valuable share.
The sharp swings of the your train and valid loss/acc curves seem abnormal.
Generally, after 400k steps of training, even for large data sets, the NAR ACC will be more than 68%.
So there may be some issues in your data pipeline?
@lifeiteng What do you think?

Thank you for your detailed and valuable share. The sharp swings of the your train and valid loss/acc curves seem abnormal. Generally, after 400k steps of training, even for large data sets, the NAR ACC will be more than 68%. So there may be some issues in your data pipeline? @lifeiteng What do you think?

I train this model on 20 different languages. So I believe it has issues handling some of these, or the dialects of a certain subset of the data. I believe it is still improving in accuracy though.
If it turns out to come about badly / inaccurate after 10 epochs still; I will stop the multilang experiment and continue with librilight english only instead.

VALL-E(this repo) focus on single language, in order to support mulit-lang, we should design the some experiments and verify them.

If this PR can get reasonable results, I'm OK to approve it.
Anyway, Thanks to @RuntimeRacer contrib.

@chenjiasheng we can synthesize some audio to judge the effectiveness of the model & data pipeline.
66% seems fine, the model haven't converge

VALL-E(this repo) focus on single language, in order to support mulit-lang, we should design the some experiments and verify them.

If this PR can get reasonable results, I'm OK to approve it. Anyway, Thanks to @RuntimeRacer contrib.

@chenjiasheng we can synthesize some audio to judge the effectiveness of the model & data pipeline. 66% seems fine, the model haven't converge

Yes it is still in the process of converging; I believe even the 10 Epoch AR Model was way from being fully converged; so it might be worth the effort to do another Follow-up training on Stage 1 / AR model.

@lifeiteng Do you think I can just continue AR training independently later despite NAR model has already been trained; and after like ~10 more Epochs AR (which would be 20 in total) do another 10 Epochs on NAR for fine-tune?

So current Iteration would be 10 AR / 10 NAR

and later 20 AR / 20 NAR for example

@RuntimeRacer NAR need more epochs than AR. You can switch to train AR.
Now you can try synthesis audio by infer.py --continoul which can verify if NAR model works.

@lifeiteng I am confused now. I tried to restart AR Training from the checkpoint I already had trained NAR on. Used this command:
python3 bin/trainer.py --max-duration 80 --filter-min-duration 0.5 --filter-max-duration 14 --train-stage 1 --num-buckets 6 --dtype bfloat16 --save-every-n 2500 --valid-interval 2500 --model-name valle --share-embedding true --norm-first true --add-prenet false --decoder-dim 1024 --nhead 16 --num-decoder-layers 12 --prefix-mode 1 --base-lr 0.05 --warmup-steps 200 --average-period 0 --num-epochs 40 --start-epoch 11 --start-batch 0 --accumulate-grad-steps 4 --world-size 6 --keep-last-k 40 --exp-dir exp/valle --manifest-dir /workspace/kajispeech-v2/commonvoice/data/tokenized --text-tokens /workspace/kajispeech-v2/commonvoice/data/tokenized/unique_text_tokens.k2symbols --oom-check false

It also said it loaded from the existing epoch-10.pt file, which contains 5 epochs of NAR.

But now after a few hours I checked Graphs and outputs, and it kinda started completely from scratch now:

All the checkpoints contain only AR weights according to size, and started off from step 0.
However the graph implicates to me that it in fact did not start completely from scratch I believe; it converges from loss 2.7 - but not fully sure:

And well, NAR weights seem to be completely erased in the new checkpoints according to size.

@RuntimeRacer
Hey, bro, please just train AR and NAR independently, using stage=1 and stage=2 respectively.
No need to rely on the tricky checkpoint transfer mechanism between AR and NAR.

@lifeiteng
How about let's disable train_stage 0, and add both AR and NAR checkpoint to args of infer.py, instead of a single merged checkpoint?
I see many people confused here, including myself two weeks ago.
Maybe @RuntimeRacer could make a PR? If not, I will.

@RuntimeRacer @chenjiasheng Yes, we can do better!

There exists a trick in current impl.,
First --train-stage 1 -> best-valid.pt
Then, cp best-valid.pt to epoch-2.pt, the train with --start-epoch 3=2+1, which reload weights trained with --train-stage 1 then start train NAR weights

PR is welcome! I didn't have time to hands on it now.

Small update on AR model training progress:

It's chewy but continues to converge. I'll keep it running now until valid loss stops converging eventually.